feat(transducer): backfill data_maturity on acoustic (Wellntel) observations - #868

Adds a data migration setting data_maturity on the 394,086 acoustic (Wellntel) transducer observations that alembic revision b2c3d4e5f6a7 left NULL. Closes the gap reported in BDMS-1169.

Why they were NULL

b2c3d4e5f6a7 backfilled maturity from nma_waterlevelscontinuous_pressure_qced, the AMPAPI flag recording whether a reading was quality controlled. Acoustic readings have no such flag — AMPAPI's WaterLevelsContinuous_Acoustic table has no QCed column at all — so every acoustic row was skipped. That is the entire Wellntel record: 394,086 rows, 54 wells, 176 deployments.

Verified against a clone of production that the two cohorts are the same set, with no overlap:

total transducer_observation                                2,582,426
  pressure_global_id NOT NULL                               2,188,340
  acoustic_global_id NOT NULL                                 394,086
  pressure_qced IS NULL                                       394,086
  pressure_qced IS NULL *and* acoustic_global_id NOT NULL      394,086
  pressure_qced IS NULL *and* acoustic_global_id NULL                0

What it does

update(TransducerObservation)
  .where(
      TransducerObservation.nma_waterlevelscontinuous_acoustic_global_id.isnot(None),
      TransducerObservation.data_maturity.is_(None),
  )
  .values(data_maturity="approved")

Matches on provenance, not on absence. acoustic_global_id is written by WaterLevelsContinuousAcousticTransferer._legacy_payload on every acoustic row and never by the pressure transferer, so it says "this row came from the acoustic table" rather than "this row has no pressure QC flag". The two happen to select the same rows today; only the first stays correct if a future loader writes rows with neither.

Only touches rows that are still NULL. Re-running is a no-op, and a maturity set deliberately since — by the hydrograph corrector, or by a later migration once the acoustic QC history is known — is left alone rather than reset to the blanket value.

The value is a recorded decision, not a derivation. There is no QC field in the acoustic legacy schema to read, so nothing here computes the answer. The docstring says so explicitly, and notes that the transfer's review_status='approved' blocks are not evidence for it — those are derived from PublicRelease, which every acoustic source row carries and which describes visibility rather than review. Worth a look from whoever owns the continuous record; BDMS-1169 is still open on whether the Wellntel data has been QCed and by what process.

alembic_revision="b2c3d4e5f6a7", so the runner refuses to execute against a database without the column.

Tests

Two added to tests/test_data_migrations.py:

• the predicate — acoustic-with-NULL updated, acoustic-with-a-value untouched, pressure-with-NULL untouched
• idempotency — two consecutive runs leave one value

MODE=development uv run pytest tests/test_data_migrations.py tests/test_data_migrations_cli.py

5 passed and 4 passed locally. MODE=development is needed because a .env with MODE=staging trips assert_auth_configuration() at import; unrelated to this change.

Not covered

Nothing in CD runs data migrations (CD_staging.yml / CD_production.yml run alembic upgrade head and stop), so merging this applies nothing. It needs oco data-migrations run by hand or a data_migrations.yml dispatch.

Could not dry-run against real data: the local production clone is at alembic b6c7d8e9f0a1, which predates data_maturity, so the column does not exist there and the runner's guard correctly rejects it. The 394,086 row count is confirmed from the provenance column, which does exist.

Related, not fixed here

While confirming the cohort, found that the continuous transfer silently drops readings — BDMS-1186 (73,113 acoustic), BDMS-1188 (318,165 pressure, ten WS- wells with no thing row), BDMS-1187 (~22k conflicting duplicate-timestamp rows). This PR sets maturity on the rows that did load; it does not load the missing ones.

🤖 Generated with Claude Code